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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SFIIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 
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Mark  Kryzsko — Mr.  Krzysko  serves  as  the  deputy  director  of  the  Enterprise  Information  and  Office  of 
the  Secretary  of  Defense  Studies.  In  this  senior  leadership  position,  he  oversees  Federally  Funded 
Research  and  Development  Centers  and  directs  data  governance,  technical  transformation,  and 
shared  services  efforts  to  make  timely,  authoritative  acquisition  information  available  to  support 
oversight  of  the  Department  of  Defense’s  major  programs — a  portfolio  totaling  more  than  $1.6  trillion 
of  investment  funds  over  the  life  cycle  of  the  programs. 

Preceding  his  current  position,  Mr.  Krzysko  served  as  ADUSD  for  business  transformation, 
providing  strategic  guidance  for  re-engineering  the  Department’s  business  system  investment 
decision-making  processes.  He  also  served  as  ADUSD  for  strategic  sourcing  &  acquisition  processes 
and  as  director  of  the  Supply  Chain  Systems  Transformation  Directorate,  championing  and  facilitating 
innovative  uses  of  information  technologies  to  improve  and  streamline  the  supply  chain  process  for 
the  Department  of  Defense.  As  the  focal  point  for  supply  chain  systems,  he  was  responsible  for 
transformation,  implementation,  and  oversight  of  enterprise  capabilities  for  the  acquisition,  logistics, 
and  procurement  communities.  In  addition,  Mr.  Krzysko  served  as  advisor  to  the  deputy  under 
secretary  of  defense  for  business  transformation  on  supply  chain  matters  and  as  the  functional 
process  proponent  to  the  Department’s  business  transformation  efforts,  resulting  in  the  establishment 
of  the  Business  Transformation  Agency. 

In  March  2002,  Mr.  Krzysko  joined  the  Defense  Procurement  and  Acquisition  Policy  office  as 
deputy  director  of  e-business.  As  the  focal  point  for  the  acquisition  domain,  he  was  responsible  for 
oversight  and  transformation  of  the  acquisition  community  into  a  strategic  business  enterprise.  This 
included  driving  the  adoption  of  e-business  practices  across  the  Department,  leading  the  move  to 
modernize  processes  and  systems,  and  managing  the  investment  review  process  and  portfolio  of 
business  systems.  Mr.  Krzysko  served  as  the  division  director  of  Electronic  Commerce  Solutions  for 
the  Naval  Air  Systems  Command  from  June  2000  to  March  2002.  From  April  1991  until  March  2000, 
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Mr.  Krzysko  served  in  various  senior-level  acquisition  positions  at  the  Naval  Air  Systems  Command, 
including  contracting  officer  of  F/A-1 8  foreign  military  sales,  F/A-18  developmental  programs,  and  the 
F-14.  In  addition,  he  served  as  program  manager  of  Partnering,  the  Acquisition  Business  Process  Re¬ 
engineering  Effort,  and  as  acquisition  program  manager  for  the  Program  Executive  Office  for  Tactical 
Aircraft. 

Mr.  Krzysko  began  his  career  in  the  private  sector  in  various  executive  and  managerial  positions, 
including  assistant  managing  director  for  Lord  &  Taylor  Department  Stores  and  operations 
administrator  for  Woodward  &  Lothrop  Department  Stores.  Mr.  Krzysko  holds  a  Bachelor  of  Science 
degree  in  finance  from  the  University  of  Maryland  University  College,  College  Park,  MD,  and  a  Master 
of  General  Administration  degree  in  financial  management  from  the  same  institution. 
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and  leads  the  System-of-Systems  Laboratory  (SoSL),  which  includes  graduate  and  undergraduate 
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Abstract 

The  complex  interdependencies  between  systems  organized  for  a  system-of-systems  (SoS) 
capability  pose  a  challenge  to  effective  acquisition  management  of  SoS  assets.  In  general, 
methodologies  to  assess  risk  that  cascades  through  interdependencies  are  critical  to 
effectively  analyzing  alternatives  in  a  capability-based  acquisition  strategy.  A  particular 
problem  occurs  in  cases  where  requirements  on  systems  are  evolving.  In  this  paper,  a 
Bayesian  Network  (BN)  method  is  presented,  which  models  requirement  evolution  in  the 
midst  of  system  interdependencies.  The  method  analyzes  the  cascading  effects  of 
requirement  and  systems  interdependencies  on  risk.  A  primary  output  of  the  approach  is  the 
identification  of  both  critical  systems  and  requirements.  A  synthetic  problem  is  solved  to 
demonstrate  the  proposed  method. 

Introduction 

Acquisition  management  continues  to  struggle  with  complex  dependencies  between 
systems  in  both  technical  and  programmatic  dimensions  in  the  face  of  requirements  that 
evolve  during  the  development  process.  The  Government  Accountability  Office  (GAO) 
studies  have  identified  the  evolution  of  requirements  as  a  source  of  program  cost  and 
schedule  overruns  (Sullivan,  2011).  Therefore,  an  adequate  assessment  of  the  cascading 
effects  of  risk  among  interdependent  systems  in  the  presence  of  evolving  requirements  has 
the  potential  to  reduce  cost  and  schedule  overruns.  The  research  in  this  paper  aims  to 
provide  a  methodology  that  supports  pre-  and  post-milestone  B  activities  by  analyzing  the 
impact  of  requirement  changes  and  system  development  failures  during  the  generation  of  a 
system-of-systems  capability. 

According  to  Maier  (1998),  a  system-of-systems  (SoS)  refers  to  a  collection  of 
geographically  distributed,  heterogeneous,  collaborative  systems  that  demonstrate 
operational  and  managerial  independences.  The  collaboration  can  be  represented  via 
networks  that  define  interactions  required  to  achieve  a  unique  capability.  Further,  in  a 
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capability-based  acquisition  setting,  requirements  are  expressed  in  terms  of  capabilities,  and 
interdependencies  between  requirements  add  another  dimension  of  complexity  to  system 
development.  Complexity  is  further  aggravated  by  the  many  stakeholders  who  influence 
requirements  and  may  even  have  conflicting  requirements  for  the  same  SoS.  An  example  of 
this  is  the  development  of  the  Joint  Strike  Fighter  (JSF),  or  F-35,  which  continues  to  face 
schedule  and  cost  overruns  (Wilson,  2011),  partially  due  to  a  development  climate  that 
includes  three  different  services  and  many  potential  international  buyers,  each  of  which  has 
differing  requirements  and  requirement  hierarchies.  Documented  case  studies  in  the 
development  of  computer-based  systems  have  shown  that  requirement  interdependencies 
are  real  and  that  they  tend  to  evolve  over  time  (Anderson  &  Felici,  2001). 

Interdependent  systems  are  directly  and  indirectly  impacted  by  the  associated 
networks  of  interdependent  requirements,  resulting  in  hierarchical  networks.  Figure  1  is  a 
simplified  adaptation  from  current  literature  on  SoS  artifacts  and  their  employment  in  the 
engineering  of  SoS  capability  architectures  using  a  wave  model  structure  (Lane,  Dahmann, 
Rebovich,  &  Lowry,  2012).  The  model  is  adapted  here  to  include  hierarchy  and  time  scale 
that  range  from  the  broad,  overarching  objectives  that  are  strategic  in  nature  (y-level)  to  the 
tactical  aspects  of  individual  system  (and  subsystem)  acquisition  (a-level).  This  paper 
focuses  on  interdependency  analysis  (i.e.,  a-level  process  in  Figure  1)  with  requirement 
evolution  from  above  (perhaps  due  to  shifts  in  portfolio  management)  occurring  during  the 
development  process. 


Figure  1.  System-of-Systems  Acquisition  Hierarchy 


Research  reported  in  this  paper  follows  from  methods  and  tools  targeting  improved 
decision-support  for  SoS  capabilities  developed  under  prior  grants  sponsored  by  the  NPS 
Acquisition  Research  Program  and  reported  in  papers  at  the  2008  (Ghose  &  DeLaurentis, 
2008),  2009  (Mane  &  DeLarentis,  2009a),  2010  (Mane  &  DeLaurentis,  2010),  and  2011 
(Mane  &  DeLaurentis,  2011)  NPS  Acquisition  Research  Symposiums.  Research  conducted 
during  the  first  year  centered  on  the  computational  exploratory  model  (CEM)  based  on  the 
16  basic  technical  management  and  technical  system-engineering  processes  outlined  in  the 
Systems  Engineering  Guide  for  System-of-Systems  (SoS-SE)  (Department  of  Defense 
[DoD],  2008)  and  Defense  Acquisition  Guidebook  (DoD,  2012).  Research  efforts  during  the 
second  year  entered  on  improvements  to  allow  for  modeling  of  scenarios  that  illustrate  the 
underlying  dynamics  that  produce  schedule  delays  and,  ultimately,  cost  overruns.  The  third 
year  of  research  focused  on  the  addition  of  system  development-risk  detail  that  enables  the 
analysis  of  the  impact  of  system  maturity  on  the  development  process  when  the  higher- 
order  effects  of  interdependencies  are  captured.  The  fourth  year  of  research  added  a 
capability  module  based  on  Markov  analysis  to  the  CEM  that  aggregate  the  network 
interdependency  characteristics  and  compare  alternatives  with  respect  to  the  time  required 
to  arrest  the  propagation  of  development  delays  in  a  network.  The  current  research  is 
focused  on  tools  suitable  to  analyzing  uncertainties  in  systems  and  the  interdependencies 
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between  systems  and  possibly  evolving  requirements.  The  rationale  is  that  such  analysis 
can  reduce  risk  to  program  success. 

A  Bayesian  Network  (BN)  is  used  to  represent  interdependencies  between  systems 
and  between  requirements  in  an  SoS  capability  development  context.  BNs  are  explored  due 
to  their  strength  in  representing  causal  relationships  between  systems  involving  uncertainty. 
Uncertainty  is  addressed  in  a  BN  using  a  continuous  density  distribution,  allowing  an 
increase  in  the  robustness  of  model  outcomes.  The  BN  is  exercised  on  a  synthetic  problem 
to  compute  time  impacts  due  to  both  requirements  evolution  and  the  impact  of 
interdependencies  between  component  systems. 

The  Analytical  Approach 

A  Hierarchical  Network  Representation  of  System-of-Systems 

To  achieve  a  wide  range  of  objectives,  each  system  operates  independently,  but 
each  must  also  interact  effectively  with  other  systems  to  meet  the  specified  SoS  capability. 
Thus,  it  is  necessary  to  understand  the  interdependencies  between  component  systems  to 
obtain  high  confidence  in  achieving  capability  levels.  As  mentioned,  components  in  an  SoS 
assemble  in  networks  and  present  interdependencies  within  a  level  and  between  levels  in  a 
hierarchy.  Figure  2  shows  the  multi-level  network  in  more  detail  that  includes  three  levels: 
capability,  requirements,  and  systems  levels  (Mane  &  DeLaurentis,  2011).  At  the 
requirement  level,  each  node  represents  a  requirement,  while  each  link  represents  the 
interdependency  between  requirements.  Likewise,  at  the  system  level,  each  node 
represents  a  system,  and  each  link  represents  the  interdependency  between  systems 
(Mane  &  DeLaurentis,  2009b).  Interdependent  systems  are  grouped  to  fulfill  a  requirement, 
while  interdependent  requirements  are  expected  to  achieve  a  capability.  The 
interdependencies  contribute  to  increasing  capability  but  also  may  lead  to  failure  through 
concealed  risks  (Mane  &  DeLaurentis,  2011).  Hence,  evaluating  the  impact  of 
interdependencies  is  a  critical  task. 


Figure  2.  Hierarchical  Multi-Level  Representation  of  a  System-of-Systems 

Capability 

An  Interdependency  Analysis  of  System-of-Systems  Using  a  Bayesian  Network 

In  this  paper,  a  BN  is  adopted  to  analyze  uncertain  interdependencies  between 
systems.  The  BN  can  graphically  represent  interactions  among  multiple  components  and 
provide  the  basic  structure  for  analyzing  and  visualizing  the  capacity-based  acquisition 
model.  The  BN  is  a  probabilistic  tool  that  evaluates  networks  of  systems  with  respect  to 
disruption  propagation  in  developing  systems  for  an  SoS.  The  evaluation  not  only  identifies 
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critical  components  from  a  risk  perspective  while  considering  requirement  revolution,  but 
also  it  can  determine  component  requirement  flexibilities  given  expected  development  time 
of  the  SoS  capability. 

A  Bayesian  Network 

A  BN  is  a  directed  acyclic  graph  (DAG)  whose  nodes  are  the  random  variables  and 
whose  edges  directly  influence  one  another.  Local  probabilities  represent  the  nature  of  the 
dependence  of  each  variable  (node)  on  its  parents.  Probability  information  in  a  BN  model  is 
defined  through  these  local  distributions.  A  node  with  no  parent  node  in  the  BN  model 
denotes  a  random  variable  and  its  associated  probability  distribution.  A  node  with  its  parent 
node(s)  can  be  represented  as  a  conditional  probability  distribution  (CPD). 

Uncertainty  with  Beta  Distribution 

Probability  information  in  a  BN  is  collected  through  experiment  tests,  measured  data, 
or  expert  opinions.  Many  errors  can  occur  in  developing  this  information.  It  is  important  to 
account  for  uncertainty  in  the  model  in  order  to  generate  reliable  results.  In  this  paper,  which 
treats  a  synthetic  proof-of-concept  example,  beta  distributions  are  used  for  node  probability 
information  to  address  uncertainties.  Beta  distributions  are  a  family  of  continuous  probability 
distributions  defined  on  the  interval  between  0  and  1  parameterized  by  two  positive  shape 
parameters  (a  and  (3).  There  are  several  reasons  to  use  a  beta  distribution.  First,  a  beta 
family  is  rich  in  shapes,  allowing  representation  of  various  types  of  probability  information 
(Reese,  Johnson,  Hama,  &  Wilson,  2005).  Figure  3  presents  various  shapes  of  beta 
distributions  according  to  two  positive  shape  parameters.  Second,  when  there  is  no 
available  probability  information,  the  beta  family  is  the  best  choice  for  use  in  determining  the 
most  conservative  probability  information. 


Figure  3.  Various  Shapes  of  Beta  Distributions 

Propagating  System  Failures  Through  Interdependencies 

There  are  two  sources  of  system  failure  in  an  SoS  context:  heritage  and  propagating. 
If  a  system  fails  on  its  own,  then  it  is  called  a  heritage  failure.  However,  if  a  system  failure  is 
caused  by  propagating  effects  from  interdependent  systems,  it  is  then  called  a  propagating 
failure.  It  is  therefore  important  to  fuse  all  failure  information,  heritage  and  propagated. 

Figure  4  shows  a  simple  BN  where  the  node  Y  has  N  parent  nodes.  This  paper 
focuses  on  binomial  failures  for  a  node.  For  example,  each  node  in  the  BN  can  only  take 
either  0  or  1  to  represent  the  status  of  the  component,  “failure”  or  “working,”  respectively. 
This  is  a  limiting  factor  of  the  approach  that  we  revisit  at  the  conclusion. 
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Figure  4.  A  Bayesian  Network  Representation 

Consider  that  each  node  has  its  own  heritage  failure  rate  defined  by  a  beta 
distribution  and  that  node  Y  has  n  parents,  X^.-Xn.  A  beta  distribution  is  parameterized  by 
two  positive  shape  parameters,  denoted  by  sn+1  and  sn+nn+1.  These  two  positive 
parameters  are  interpreted  as  the  number  of  failures  and  survivors,  respectively,  when  sn 
and  nn  are  integers  (Spring  &  Thompson,  1966).  Let  PA(Y)  denote  the  set  of  the  parents  of 
the  node  Y  (i.e.,{  X^-.Xn }).  According  to  the  law  of  total  probability,  the  propagating  failure 
rate  of  node  Y  is 


p(Y  =  0)  =  X  CPDkP{PA(Y )  =  k)  (1 ) 

k 

where  CPDk  =  p(Y  =  0| PA(Y)  =  k) ,  k  is  all  combination  of  parent  node  values.  For 

example,  if  PA(Y)  includes  two  parent  nodes  (Xi  and  X2),  then  k  =  {{0,0},  {0,1},  {1,0},  {1,1}} . 
Therefore,  a  node  with  two  parent  nodes  has  four  CPDk  values.  CPDk  values  here  indicate 
the  dependency  strength  of  a  failure  propagating  to  a  dependent  system.  For  instance,  if 
node  X1  (or  X2)  fails,  then  node  Y  has  a  30%  (or  20%)  chance  to  fail  by  a  propagating  effect 
of  the  node  X!  (or  X2)  failure.  In  this  case,  all  CPDk  values  are  determined: 
p(Y  =  0|X,  =1,  X2=0)  =  0.3  ,  p{Y  =  0|X,  =  0,  X2  =  1)  =  0.2 ,  p(Y  =  0|X,  =  0,  X2  =  0)  =  0 ,  and 

p(Y  =  o|x,  =  l,  X2  =  l)  =  0.5  ■  Analytical  solution  p{Y = 0)for  the  propagating  failure  rate  of  node 

Y  is  very  likely  to  be  non-parametric  due  to  its  complicated  functional  form.  For 
computational  convenience,  we  use  the  approach  in  reference  (Thompson  &  Haynes,  1980; 
Liu,  Li,  &  Kim,  201 1 )  to  approximate  the  propagating  failure  rate  with  a  beta  distribution 
having  the  same  first  two  moments.  Let  beta(b,c)  denote  the  beta  distribution  of  the 
propagating  failure  rate  of  node  Y.  Then,  the  first  two  moments  of  this  distribution  are 

M.  =  E(Y )  =  -^—,  and  M2  =  E(Y2)  =  —  E(Y)  (2) 

b+c  b+c+ 1 

The  first  moment  of  p(Y=0)  is  the  mean: 

M,  =  E(p(Y  =  0))  =  E  (X  CP Dk  p(P A(Y)  =  k))  =  ^CPD^EipiPA^Y)  =  j )] 

k  k  '  (3) 

where  j  is  the  value  for  PA(Y)  in  the  set  of  k.  For  computational  ease,  Equation  3  can  be 
further  written  as  follows: 


M,  =  E(p(Y  =  0))  =  X  CPDk  n  {j,  ~  E[p(PA1  (7)  =  0)] }  =  X  CPDk  j 


Sj  +1 
n,  +  2 


The  second  moment  of  p(Y=0)  is  the  mean  of  p(Y=0)2: 

M2  =  E(p(Y  =  0)2 )  =  Y,CPDk  n{y',2 -2J,^L  +  ^+X} 
k  i  {  n,+  2  («,-  +2)(«,.  +3)  J 

Finally,  we  can  define  two  parameters,  b  and  c,  for  a  beta  distribution  of  p(Y=0)  as 


(4) 

(5) 


ru.i  i,i  nti  v  t?.]  iLm 
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6_M,(M,-M2)  „_(1  -M.XM'-M,)  (6) 

(M2  -Mj2)  ’  (M2  -M,2) 

Now,  node  Y  has  two  different  beta  distributions,  one  for  heritage  failure  rate  and  one 
for  propagating  failure  rates  encapsulated  in  Equation  6.  These  two  beta  distributions  are 
integrated  to  get  the  new  failure  rate  distribution  of  node  Y,  including  both  failure 
information:  heritage  and  propagating.  This  task  can  be  easily  completed  using  the  same 
process  for  obtaining  fusion  of  all  propagating  failure  information  mentioned  above  with 
different  CPDs  indicating  100%  propagating  effects.  After  applying  these  two  fusion 
processes  (the  first  is  fusion  of  propagating  effects  from  dependent  systems  for  the 
propagating  failure  rate,  and  the  second  is  fusion  of  both  heritage  and  propagating  failure 
rates  for  the  new  integrated  failure  rate)  to  all  nodes  in  a  network,  we  can  obtain  the  beta 
distributions  of  the  new  failure  rate  information,  including  propagating  effects  for  all  nodes. 

This  result  can  be  used  to  determine  the  critical  component  indicating  the  vulnerable 
component  and  expected  development  time  for  a  whole  SoS. 

A  Representation  of  Requirement  Evolution  with  Technology  Readiness  Level 

Requirement  evolution  refers  to  changes  that  take  place  in  a  set  of  system 
requirements  after  the  initial  requirement  analysis  (Anderson  &  Felici,  2001).  In  the  SoS 
capability  context,  requirement  evolution  seems  inevitable.  For  instance,  an  initial 
performance  goal  for  payload  capacity  of  an  aircraft  is  250,000  lbs,  while  in  the  future, 
customers  might  ask  for  300,000  lbs.  Or  decision-makers  may  decide  to  reduce  the  payload 
capacity  to  200,000  lbs  due  to  the  considerable  delay  and  cost.  Along  with  requirements 
evolving,  failure  rates  of  components  related  to  requirement  evolution  may  change  as  well. 

Technology  readiness  levels  (TRLs)  are  a  systematic  metric/measurement  NASA 
invented  that  supports  assessments  of  the  maturity  of  a  particular  technology  and  the 
consistent  comparison  of  maturity  between  different  types  of  technology  (Mankins,  1995).  In 
other  words,  TRLs  represent  the  current  highest  available  technology  level,  which  is  also  a 
time-dependent  variable.  Thus,  both  requirement  evolution  and  TRLs  can  impact  component 
system  failure  rates.  Take  the  previous  aircraft  payload  capacity  for  example.  In  the  former 
case,  although  we  require  a  higher  requirement  capability,  if  TRL  is  increasing,  meaning  that 
the  technology  has  been  continuously  proven  for  an  application,  at  the  same  time  failure 
rates  might  decrease,  keep  the  same,  or  still  increase.  Similarly,  in  the  latter  case,  TRL  may 
influence  failure  rates  by  reducing  a  certain  amount  of  its  value.  Additionally,  TRLs  could 
effect  requirement  evolutions.  For  example,  policy  makers  may  consider  technology 
readiness  levels  and  current  development  state  together  to  decide  whether  necessary  to 
reduce  requirement  capability. 

The  critical  connection  between  requirement  evolution,  TRL,  and  component  failure 
rates  plays  an  essential  part  in  the  development  of  the  SoS  capability.  In  the  synthetic 
problem  presented  in  the  Research  Result  section,  we  assume  TRL  to  be  constant  for  all 
component  systems.  Therefore,  if  a  requirement  evolves  to  a  higher  level,  system  failure 
rates  always  increase. 

Research  Result:  A  Synthetic  Demonstration  Problem 

A  simple  synthetic  problem  is  formulated  and  solved  to  demonstrate  the  proposed 
BN  approach.  Figure  5  shows  the  representation  of  a  five-system  network  where,  for 
example,  the  development  of  system  1 ,  here  denoted  by  S-i,  depends  on  the  development  of 
systems  2  and  4.  This  implies  that  information  from  one  system  development  process 
affects  the  development  of  dependent  systems.  For  system  1 ,  information  flows  from  system 
2  and  system  4  to  system  1  to  fuse  all  information  from  dependent  systems.  The  T  values 
indicate  the  dependency  strength  and  correspond  to  the  conditional  probability  of  a  failure 
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propagating  to  a  dependent  system.  For  instance,  if  system  4  fails,  then  system  1  has  a 
30%  chance  to  fail  by  a  propagating  effect  of  the  system  4  failure.  The  table  in  Figure  5 
summarizes  heritage  failure  rates  for  all  systems  in  terms  of  beta  distributions,  means,  and 
standard  deviations. 


System 

Failure  Rate 
Distribution 

Mean  of 
Failure  Rate 

Standard  Deviation 
of  Failure  Rate 

System  1 

Beta  (20, 80) 

0.2 

0.04 

System  2 

Beta  (5, 95) 

0.05 

0.02 

System  3 

Beta  (25, 75) 

0.25 

0.04 

System  4 

Beta  (10, 90) 

0.1 

0.03 

System  5 

Beta  (30, 70) 

0.3 

0.05 

Figure  5.  A  Five-System  Development  Network  and  Heritage  Failure  Rate 

Information 

Consider  the  information  fusion  of  failure  rates  of  systems  2  and  3  with  system  5 
(see  Figure  6).  System  5  is  connected  to  two  dependent  systems,  2  and  3,  with 
interdependency  strength  of  0.1  and  0.25,  respectively.  Systems  2  and  3  have  their  own 
heritage  failure  rates,  with  beta  distributions  shown  in  Figure  6a.  The  propagating  failure  rate 
on  system  5  is  easily  calculated  through  the  proposed  approach  in  this  paper,  based  on  the 
given  information  about  heritage  failure  rates  and  conditional  probability  for  propagating 
effects.  In  Figure  6,  blue  lines  denote  the  heritage  failure  rate  distributions  for  systems,  and 
red  lines  denote  the  propagating  failure  rate  on  system  5,  respectively.  The  green  line  in 
Figure  6b  represents  the  integrated  failure  rate  for  system  5.  The  mean  of  the  system  5 
integrated  failure  rate  represents  an  increase  of  0.07  over  its  heritage  rate  value  due  to  the 
propagating  effects  from  dependent  system  failures.  Figure  7  shows  the  mean  values  of 
propagating  effects  for  all  systems.  These  values  depend  on  the  number  of  dependent 
systems  and  the  failure  rates  of  dependent  systems.  System  2  has  the  highest  propagating 
effects,  indicating  strong  dependencies  with  numerous  other  systems.  It  also  has  a  higher 
probability  to  be  disrupted  by  other  system  failures  during  the  development  process. 

Because  it  is  hard  to  control  this  kind  of  failure,  the  design  team  for  system  2  should 
consider  these  propagating  effects  when  scheduling  for  the  development  time. 
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(a)  Propagating  failure  rate  (b)  Integrated  failure 

Figure  6.  Information  Fusion  of  Failure  Rates  of  Systems  2  and  3  with 
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Figure  7.  Mean  Values  for  Failure  Rates  Propagating  to  Systems 

The  same  synthetic  problem  is  now  considered  with  the  requirement  evolution 
modeled  via  modified  heritage  failure  rates.  The  rationale  is  that,  when  a  system’s 
requirements  become  more  stringent  in  efforts  to  reach  a  new  capability  goal,  an  increase  of 
heritage  failure  rate  follows.  The  total  expected  development  time  is  adopted  to  measure 
development  time  of  an  SoS  capability.  We  assumed  that  the  expected  development  time 
for  each  system  is  1 ,  indicating  that  if  there  is  no  failure,  each  system  can  be  done  in  one 
time  unit.  Every  system  also  has  development  delay  time  due  to  its  failure  rate,  calculated  as 
the  product  of  failure  rate  and  the  expected  development  time.  For  instance,  if  a  system’s 
failure  rate  is  0.6  and  expected  development  time  for  the  system  is  1 ,  then  the  design  team 
of  this  system  needs  1 .6  more  times  than  normal  to  complete  it.  Therefore,  the  total 
expected  development  time  can  be  formulated  as  the  follows: 


Total  expected  developmert  time  =  ^(l  +  failure  ratet  x expected developmert  timel ,)  (7) 

i 

We  increase  the  heritage  failure  rate  of  each  system,  one  at  a  time,  from  0  to  0.5. 

The  resulting  expected  time  to  develop  the  systems  to  meet  the  SoS  capability  in  each  case 
is  shown  in  Figure  8.  System  1  has  the  steepest  slope,  indicating  that  it  is  most  critical  in 
terms  of  impact  on  total  expected  development  time.  The  total  expected  development  time 
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for  all  systems  evolves  linearly,  a  result  of  the  model  assumption  of  constant  interdependent 
strengths. 


Figure  8.  Evolution  of  Total  Expected  Development  With  Respect  to  Failure  Rate 

Increases 


However,  if  decision-makers  knew  how  much  requirement  change  could  be  tolerated 
within  the  budgeted  total  development  time,  then  they  could  make  more  informed  choices.  In 
the  present  method,  this  is  done  using  a  system  upgrade  capacity  diagram.  Figure  9  shows 
the  system  upgrade  capacity  diagram  for  the  synthetic  problem.  The  total  expected 
development  time  is  set  to  a  limit  of  7,  and  the  maximum  increase  of  failure  rate  for  each 
system  is  computed.  System  4’s  upgrade  capacity  is  higher  than  others,  meaning  that  it  can 
be  substituted  with  an  alternative  system,  which  has  higher  failure  rates.  However,  system  1 
has  the  lowest  upgrade  capacity  because  it  was  the  most  critical  system  in  the  synthetic 
problem,  according  to  Figure  8. 


Figure  9.  System  Upgrade  Capacity  for  Requirement  Evolutions 


Conclusion 

The  development  process  of  an  SoS  capability  is  often  affected  by  risks  from 
interdependencies  between  constituent  systems  and  the  requirement  evolution  during  the 
development  life  cycle.  This  paper  adopts  a  BN  approach  to  analyze  interdependencies  by 
measuring  component  failure  rates  and  total  expected  development  time  for  a  whole  SoS. 
Propagating  failure  rates  are  calculated  to  describe  interdependencies,  with  heritage  failure 
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rates  being  evaluated  for  individual  systems.  By  the  integration  of  these  two  failure  rates, 
both  currently  expressed  in  beta  distributions,  a  new  failure  rate  distribution  is  achieved  that 
more  completely  represents  the  true  risk  and  more  faithfully  determines  the  critical 
components. 

A  simple,  synthetic  five-system  SoS  problem  demonstrates  the  proposed  BN 
approach.  Results  illustrate  that  the  integrated  failure  rate  for  individual  systems  increases 
due  to  the  propagating  effects  of  interdependencies.  The  comparison  of  integrated  failure 
rates  among  all  systems  is  useful  in  identifying  the  most  critical  system.  Meanwhile,  given  a 
development  time  constraint,  upgrade  capacity  for  each  system  when  requirements  evolves 
were  computed. 

The  specific  BN  formulation  approach  in  this  paper  rests  on  two  basic  assumptions. 
First,  the  failure  rates  (selected  based  on  TRL)  are  assumed  constant.  Second,  the 
interdependency  strengths  between  systems  are  assumed  constant.  Hence,  future  work  will 
address  the  relaxation  of  the  assumptions.  More  generally,  two  additional  challenges  must 
be  addressed.  First,  a  BN  approach  can  use  only  0  or  1  to  represent  two  discrete  states,  like 
“working”  or  “failure”;  thus,  continuous  variables,  such  as  development  percentage,  cannot 
be  expressed  directly.  The  second  challenge  is  the  collection  of  input  failure  rates,  which 
were  simply  generated  arbitrarily  in  this  paper.  Such  rates  may  be  inferred  from  analyses  of 
historical  data  on  TRL  levels  as  they  related  to  eventual  development  success. 
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Motivation  —  Interdependency  in  SoS 

•  Interdependency  between  systems  is  a  necessary  characteristic  to  achieve  a 
SoS  capability 

•  Interdependency  (expressed  in  developmental  and  operational  architectures), 
however,  brings  possibility  of  risk,  especially  from  cascading  failures. 


Image  from:  http://www.nws.noaa.gov/nextgen/ngl01.shtml 
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http  ://www.mda.mil/ system/  system.html 
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Brief  History:  Our*  Methods  Development  in  this  Arena 

*  Thanks  to  NPS  ARP  and  now  also  DoD  SERC 
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Aim—  Tools  for  Better  Acquisition 


•  Aim:  Method  that  supports  pre-  and  post-  milestone  A,  B  activities  by 
analyzing  the  impact  of  requirement  changes  and  system  development 
failure  during  generation  of  a  SoS  capability. 

•  uncertainties  in  systems  and  requirement  interdependencies 

•  Evidence:  DoD,  GAO  and  others  identify  requirement  evolution  & 
technology  uncertainty  as  critical  issue 
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A  hierarchical  representation  of  an  SoS 

•  Interdependent  systems  are  grouped  to  fulfill  a  requirement  while 
interdependent  requirements  are  expected  to  achieve  a  capability. 

•  Hierarchy  in  interdependencies  contribute  to  increasing  capability  but  also 
may  lead  to  failure  through  concealed  risks 


Capability 
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Interdependency  analysis  via  Bayesian  Network  (BN) 

•  In  analyzing  interdependencies,  we  need: 

>  Inherent  uncertainty  of  systems 

>  Propagation  of  uncertainty 

•  BN  is  a  directed  acyclic  graph. 

•  Node  R  has  n  parent  nodes. 

>  R :  the  requirement 

>  Sf.  the  system  that  meet  this  requirement. 

>  PA(Rj)  denote  the  set  of  parents  of  the  node  Rp  i.e.  {Sj...Sn} 

•  Applying  the  law  of  total  probability,  the  probability  of  achieving  a  particular 
requirement,  node  Rt  is: 

P(Ri  =  1)  =  Z  P(R.  =  1 1  PA(R.  )  =  S^piPAiR, )  =  S) 

s 

where  S  is  the  set  of  all  the  possible  combinations  of  parent  node  values 

•  For  example,  if  PA(Ri)  includes  two  parent  nodes  S1  and  S2 , 
thenS={(0,  0),  (0,1),  (1,0),  (1,1)}. 
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Uncertainty  with  Beta  Distribution 

•  Beta  distributions  are  often  used  as  node  failure  probability  information 
to  address  uncertainties. 

•  Beta  distributions  are  a  family  of  continuous  probability  distributions 
defined  on  the  interval  between  0  and  1  parameterized  by  two  positive 
shape  parameters  (a  and  (3) 
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Overview  of  interdependency  analysis 

•  Each  system  has  its  own  inherent  information  (e.g.,  failure  rate  or 
reliability). 

•  Integrated  information  (e.g.,  failure  rate  or  reliability)  can  be  obtained  by 
combing  inherent  information  and  propagating  effects  from  dependent 
systems. 

•  Process: 

1 .  Estimate  propagating  effects  from  dependent  systems  using  a 
joint  probability. 

2.  Combine  the  inherent  information  and  propagating  effects. 
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Requirement  Evolution 

•  Both  requirement  evolution  and  Technology  Readiness  Levels  (TRLs)  can 
impact  component  system  failure  rates. 

>  Requirement  evolution  refers  to  changes  that  take  place  in  a  set  of  system 
requirement  after  the  initial  requirement  analysis  [1]. 

>  TRLs  are  a  systematic  metric/measurement,  invented  by  NASA  that  supports 
assessments  of  the  maturity  of  a  particular  technology  and  the  consistent 
comparison  of  maturity  between  different  types  of  technology  [2]. 

•  In  this  study,  we  assume  TRL  to  be  constant  for  all  component  systems. 

•  Therefore,  if  a  requirement  evolves  to  a  higher  level,  system  failure  rates 
will  increase  always. 

[1]  Anderson,  S.,  &  Felici,  M.  (2001).  Requirements  Evolution  From  Process  to  Product  Oriented  Management.  3rd  International 

Conference  on  Product  Focused  Software  Process  Improvement  (pp.  27-41 ).  Kaiserslautern,  Germany. 

[2]  Mankins,  J.  C.  (1995).  Technology  Readiness  Levels:  A  white  paper.  Advance  Concepts  Office,  Office  of  Space  Access  and 

Technology.  NASA. 
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Synthetic  Problem  Demonstration 

•  In  a  five-system  network,  the  development  of  system  1,  here  denoted  by 
S1?  depends  on  the  development  of  system  2  and  4. 

’  Table  below  summarizes  inherent  failure  rates  for  all  systems  in  terms  of 
beta  distributions,  means,  and  standard  deviations. 

’  T  values  indicate  the  dependency  strength  and  correspond  to  the 
conditional  probability  of  a  failure  propagating  to  a  dependent  system. 


T12=0.2 


T23=0.15 


System 

Failure  Rate 
Distribution 

Mean  of 
Failure  Rate 

Standard  Deviation 
of  Failure  Rate 

System  1 

Beta  (20,  80) 

0.2 

0.04 

System  2 

Beta  (5,  95) 

0.05 

0.02 

System  3 

Beta  (25,  75) 

0.25 

0.04 

System  4 

Beta  (10,  90) 

0.1 

0.03 

System  5 

Beta  (30,  70) 

0.3 

0.05 
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Probability  Density  Function 


Results  —  Fusion  of  failure  rate  information 


•  Figure  in  left  shows  the  fusion  of  inherent  failure  rates  for  calculate 
propagating  effects. 

•  Figure  in  right  shows  the  fusion  of  propagating  effects  with  inherent 
failure  rate  of  system  5. 


5/16/2012 


NPS  gth  Annual  Acquisition  Research  Symposium 


11 


Results  —  Comparison  of  propagating  effects 


Figure  below  shows  the  mean  values  of  propagating  effects  for  all  systems. 

System  2  has  the  highest  propagating  effects  indicating  strong 
dependencies  with  numerous  other  systems. 

It  also  has  a  higher  probability  to  be  disrupted  by  other  system  failures 
during  the  development  process. 
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Results  —  Requirement  Evolution  Effects 

•  Figure  below  shows  total  expected  development  time  of  each  system  with 
inherent  failure  rates  increasing  from  0  to  0.5. 

•  System  1  has  the  steepest  slope  --  most  critical  in  terms  of  impact  on  total 
expected  development  time. 

•  Linearity  —  a  result  of  the  model  assumption  of  constant  interdependent 
strengths. 
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Results  —Requirement  Evolution  Effects  (Cont.) 


•  Figure  below  shows  the  system  upgrade  capacity  diagram  for  the  synthetic 
problem. 

•  System  4’s  upgrade  capacity  is  higher  than  others  —  it  can  be  substituted 
with  an  alternative  system  which  has  higher  failure  rates. 


System  1  has  the  lowest  upgrade  capacity  because  it  was  the  most  critical 
system. 
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Implementation  Example 

•  Littoral  Combat  Ship  (LCS)  systems  is  adopted  as  example  for  applying  the 
proposed  approach. 

•  LCS  systems  is  designed  to  counter  growing  potential  threats  in  the  littoral 
area  such  as  coastal  mines,  quiet  diesel  submarines  and  terrorists  on  small, 


fast,  armed  boats. 


LCS  Concept  of  Operations 


Image  from:  Presentation  slides  by 
RDML  Vic  Guillory  of  OPNAV  at  Mine 
Warfare  Association  Conference  (titled 
“Littoral  Combat  Ship”,  08-May-07) 


Diesel  I  Electric  Submarines 


Data  Sharing: 
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Hierarchical  representation  of  Littoral  Combat  Ship 


MH-60R:  Armed  Helicopter  for  Surveillance  and  Attack 

MH-60S:  Helicopter  for  Airborne  Mine  Counter-Measure 

UAV:  Unmanned  Air  Vehicle 

USV:  Unmanned  Surface  Vehicle 

RMS:  Remote  Mine  Hunting  System 

Torpedo,  Missile,  and  the  LCS 


Req.  1  (Mine) 


Req.  2  (Anti-  Submarine) 


Req.  3  (Anti-Surface) 
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Interdependency  analysis 


Fusion  all  information  from  requirements 

p{C  =  \)  =  YJPi.C  =  \\  PA(C)  =  R)p(PA(C )  =  R) 

R 


Fusion  all  information  from  systems 

p(Rk  =  1)  =  £>(**  =  1 1  PA(Rt )  =  S)p(PA(R, )  =  5) 

5 


System  failure  rates 

P {System  f=  Failure )  =  1  -  e~X[t 


The  performance  of  LCS  systems  is  defined  as  the 
probability  to  complete  the  mission. 


-*7 
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Two  different  architectures 


MH-60R  MH-60R 


RMS  RMS 


-  architecture  1  has  NO  communication  between  LCS  command  centers 

-  architecture  2  DOES  have  communication  between  LCS  command  centers 
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Resilience  Metric 

•  Resilience  pattern  represents  the  relationship  between  performance  (or 
capability)  and  the  level  of  component  failures  in  the  entire  domain. 

•  The  yellow  horizontal  line  represents  the  acceptable  level  of 
performance  to  meet  a  system  requirement. 

•  The  area  of  A,  called  system  resilience  capability,  indicates  the  ability  of 
the  system  to  sustain  the  performance  beyond  a  desired  level  in  the  face 
of  failures. 


%  of  system  failures 
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Two  factors  affecting  resilience  patterns 

•  Architecture  type:  the  fundamental  organization  of  an  SoS 
embodied  by  its  components  and  their  relationships  to  each  other 
(i.e.,  interdependencies). 

•  Risks  to  SoS:  the  sources  and  consequence  of  component  failures. 
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Results  of  LCS  systems  case  study 


Evolution  of  system  performance  Static  resilience  with  one  entity 

of  both  architectures  for  600  mins  failure  in  architecture  1 


•  The  exponential  distribution  is  a 
function  of  time:  the  failure  rate  of 
a  system  increases  as  time  elapses. 

•  Arch2  has  the  better  resilience 
pattern  than  Archl . 
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•  The  systems  with  lower  static 
resilience  values  are  critical  systems. 

•  The  right  figure  indicates  that  LCS 
nodes  are  most  critical. 
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Results  of  LCS  systems  case  study 

*  Best-case:  when  critical  systems  fail  last. 

*  Worst-case:  when  critical  systems  fail  first. 

1  Expected-case:  when  system  failures  occur  based  on  systems’  own  failure  rates. 
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The  two  major  factors,  architecture  type  and  failure  rate  of  a  system,  affect  system  resilience 
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Strengths  and  weaknesses  of  the  BN 

•  Strengths 

-  Can  handle  directed  nets  of  interdependencies 

-  Can  easily  represent  the  system  structure  and  interactions 

-  Can  be  automatically  updated  with  new  observations 

-  Can  be  used  constructively  in  sensitivity  mode  when  very 
few  or  no  data  available,  with  some  prior  knowledge 

•  Weaknesses 

-  Cannot  handle  cycles  in  network. 

-  Require  rich  collection  of  data  (failure  and  performance)  to 
obtain  conditional  probability  distribution  connectivity  and 
node  information 
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Conclusion 

•  A  Bayesian  Network  approach  is  adopted  to  analyze  interdependencies  by 
measuring  component  failure  rates  and  total  expected  development  time  for  a 
whole  SoS. 

•  Integration  of  inherent  failure  rates  and  propagating  failure  rates  is  calculated  to 
more  completely  represent  the  true  risk  and  more  faithfully  determine  the  critical 
components. 

•  Upgrade  capacity  for  each  system  can  be  obtained  for  decision  makers  when 
requirements  evolve. 

Future  Work 

•  A  Bayesian  Network  approach  can  only  use  0  or  1  to  represent  two  discrete 
states,  like  ‘working’  or  ‘failure’;  thus  continuous  variables  such  as  development 
percentage  cannot  be  expressed  directly 

•  Collection  of  input  failure  rates  —  may  be  inferred  from  analysis  of  historical 
data  on  TRL  levels  as  they  related  to  eventual  development  success. 
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Thank  you 

Questions  ? 
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Overview  of  interdependency  analysis 

■  Each  system  has  its  own  inherent  information  (e.g.?  failure  rate  or 
reliability). 

■  Integrated  information  (e.g.,  failure  rate  or  reliability)  can  be  obtained  by 
combing  inherent  information  and  propagating  effects  from  dependent 
systems. 

Inherent  failure  rate  of  node  Y 


K  & 


Propagating  effects  from 
nodeXj  andX2 


Integrated  failure  rate 
of  node  Y 
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Previous  Work 


•  2008  —  Computational  Exploratory  Model  (CEM)  based  on  the  16  basic 
technical  management  and  technical  system-engineering  processes. 

•  2009  —  Improvements  to  allow  for  modeling  of  scenarios  that  illustrate 
the  underlying  dynamics  that  produce  schedule  delays  and  cost 
overruns. 

•  2010  —  Addition  of  system  development-risk  detail  that  enables  the 
analysis  of  the  impact  of  system  maturity  on  the  development  process 
when  the  higher-order  effects  of  interdependencies  are  captured. 

•  2011  —  A  capability  module  based  on  Markov  analysis  to  the  CEM  that 
aggregate  the  network  interdependency  characteristics  and  compare 
alternatives  with  respect  to  the  time  required  to  arrest  the  propagation  of 
development  delays  in  a  network. 
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Case  study  -  Littoral  Combat  Ship  (LCS)  (Cont.) 


•  The  failure  rates  are  also  given  to  all  entities  based  on  mission 
time  limit  of  that  entities  using  the  exponential  distribution. 

1  = - - - 

mission  time  limit 

•  where  T is  the  system  exposure  time  and  X  is  the  system’s 
failure  rate  which  can  be  written  as  a  function  of  the  mean  time 
between  failures  (mission  time  limit). 

•  Expected  reliability  of  LCS  systems  to  finish  the  mission  is 
calculated  as  the  performance  using  Bayesian  Network  with 
two  different  architectures:  unavailable  (archl)/available 
(arch2)  communication  between  command  centers  of  two 
Littoral  Combat  Ships. 


5/16/2012 


NPS  gth  Annual  Acquisition  Research  Symposium 


29 


Static  resilience  of  a  complex  system 


•  Two  metrics  are  defined  to  quantify  system  resilience  in  the  multi-level 
system  tester  module. 

•  The  static  resilience  can  be  thoughts  of  as  a  ‘specific’  performance  measure 
-  for  a  given  number  of  component  failures,  how  much  performance  is 
maintained. 

•  Static  resilience  is  defined  as  the  ratio  of  percentage  of  performance  of  a 
complex  system  (e.g.,  SoS)  to  percentage  of  component  failures 

%  of  performance  (or  capability) of  a  system 

Static  Resilience  =  - - — - - — - 

%  of  component  failures 
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Metric:  System  Resilience 

•  The  general  definition  of  resilience  is  the  capacity  of  a  system 
to  survive,  adapt  and  grow  in  the  face  of  change  and 
uncertainty1 . 

•  The  concept  of  resilience  has  been  used  to  analyze  systems  and 
solve  problems  in  fields  such  as  ecology,  psychology,  computer 
science,  material  science,  and  disaster  management2’3’4’5’6. 

•  See  back-up  slide  for  definition  in  different  domains 


1Fiksel,  Joseph,  “Sustainability  and  resilience:  toward  a  systems  approach,”  Sustainability:  Science,  Practice,  &  Policy,  Vol.  2,  No.  2,  pp.  1-8,  2006 

2Folke,  C.,  Carpenter,  S.,  Walker,  B.,  Scheffer,  M.,  Elmqvist,  T.,  Gunderson,  L.,  Holling,  C.S.,  "Regime  Shifts,  Resilience,  and  Biodiversity  in  Ecosystem 
Management",  Annual  Review  of  Ecology,  Evolution,  and  Systematics  Vol.  35:  557-581 , 2004 

3Masten,  A.  S,  "Ordinary  Magic:  Lessons  from  research  on  resilience  in  human  development"  (PDF).  Education  Canada  49  (3):  28-32,  2009 

4Mohammad,  A.  J.,  Hutchison,  D.,  Sterbenz,  James  P.G.  "Poster:  Towards  Quantifying  Metrics  for  Resilient  and  Survivable  Networks",  in  14th  IEEE 
International  Conference  on  Network  Protocols  (ICNP  2006),  Santa  Barbara,  California,  USA,  November  2006 

5From:  Avallone,  Eugene  A.,  et  al.  eds.  Marks'  Standard  Handbook  for  Mechanical  Engineers,  1 1th  ed.,  2007 

6Shinozuka,  M.,  Chang,  S.E.,  “Evaluating  the  Disaster  Resilience  of  Power  Networks  and  Grids,”  Springer-Verlag,  edited  by  Okuyama,  Y.,  Chang,  S.E., 
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Definitions  of  resilience  in  different  domains 


Domains 

The  definition  of  resilience 

Ecology 

The  capacity  of  an  ecosystem  to  respond  to  a  perturbation  or 
disturbance  by  resisting  damage  and  recovering  quickly.  Such 
perturbations  and  disturbances  can  include  stochastic  events  such  as 
fires,  flooding,  windstorms,  insect  population  explosions,  and  human 
activities  such  as  deforestation  and  the  introduction  of  exotic  plant  or 
animal  species. 

Psychology 

The  idea  of  an  individual's  tendency  to  cope  with  stress  and 
adversity.  This  coping  may  result  in  the  individual  “bouncing  back”  to 
a  previous  state  of  normal  functioning,  or  using  the  experience  of 
exposure  to  adversity  to  produce  a  “steeling  effect”  and  function 
better  than  expected. 

Computer  science 

The  ability  to  provide  and  maintain  an  acceptable  level  of  service  in 
the  face  of  faults  and  challenges  to  normal  operation. 

Material  science 

The  property  of  a  material  to  absorb  energy  when  it  is  deformed 
elastically  and  then,  upon  unloading  to  have  this  energy  recovered.  In 
other  words,  it  is  the  maximum  energy  per  unit  volume  that  can  be 
elastically  stored. 

Disaster 

application 

The  ability  of  countries,  communities  and  households  to  manage 
change,  by  maintaining  or  transforming  living  standards  in  the  face  of 
shocks  or  stresses  -  such  as  earthquakes,  drought  or  violent  conflict  - 
without  compromising  their  long-term  prospects. 
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Presentation  Outline 


•  Motivation  and  Research  Objectives 

•  Previous  Work 

•  Analytical  Approach 

>  A  hierarchical  representation  of  System  of  Systems  (SoS) 

>  An  interdependency  analysis  of  an  SoS  using  a  Bayesian 
Network  with  beta  distribution 

•  Synthetic  Problem  Demonstration 

•  Conclusion  &  Future  Work 
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Research  Objectives 


•  This  paper  aims  to  provide  a  methodology  that  supports  pre-  and  post¬ 
milestone  B  activities  by  analyzing  the  impact  of  requirement  changes  and 
system  development  failure  during  generation  of  a  system-of-systems 
capability. 

•  In  other  words,  this  study  is  focused  on  tools  suitable  to  analyze 
uncertainties  in  systems  and  the  interdependencies  between  systems  and 
possibly  evolving  requirements. 
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An  interdependency  analysis  using  a  BN 

•  In  analyzing  interdependencies,  we  need  to  handle: 

>  Inherent  uncertainty  of  systems 

>  Propagation  of  uncertainty 

•  A  Bayesian  Network  (BN)  can  handle  these. 

•  A  Bayesian  Network  (BN)  is  a  directed  acyclic  graph. 

>  Nodes  are  the  random  variables  and  edges  correspond  to  direct 
influence  of  one  node  on  another. 

>  Each  variable  in  the  BN  model  is  associated  with  a  conditional 
probability  distribution. 

•  A  BN  facilitates  the  formation  of  a  joint  probability: 


n 


where  Pa(X. )  are  the  parents  of  X-r 
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